Video Frame Interpolation via Adaptive Separable Convolution

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.Comment: ICCV 2017, http://graphics.cs.pdx.edu/project/sepconv

Motion-Adjustable Neural Implicit Video Representation

Implicit neural representation (INR) has been successful in representing static images. Contemporary image-based INR, with the use of Fourier-based positional encoding, can be viewed as a mapping from sinusoidal patterns with different frequencies to image content. Inspired by that view, we hypothesize that it is possible to generate temporally varying content with a single image-based INR model by displacing its input sinusoidal patterns over time. By exploiting the relation between the phase information in sinusoidal functions and their displacements, we incorporate into the conventional image-based INR model a phase-varying positional encoding module, and couple it with a phase-shift generation module that determines the phase-shift values at each frame. The model is trained end-to-end on a video to jointly determine the phase-shift values at each time with the mapping from the phase-shifted sinusoidal functions to the corresponding frame, enabling an implicit video representation. Experiments on a wide range of videos suggest that such a model is capable of learning to interpret phase-varying positional embeddings into the corresponding time-varying content. More importantly, we found that the learned phase-shift vectors tend to capture meaningful temporal and motion information from the video. In particular, manipulating the phase-shift vectors induces meaningful changes in the temporal dynamics of the resulting video, enabling non-trivial temporal and motion editing effects such as temporal interpolation, motion magnification, motion smoothing, and video loop detection

Smart City, Citizen Engagement, and Information System Research

The paper highlights the importance of involvement of citizens in all the steps of smart city initiatives.Therefore, authors try to identify key factors and enablers for effective engagement and involvement of citizens and residents in any smart city project

MACHINE LEARNING IN THE DESIGN SPACE EXPLORATION OF TSN NETWORKS

Real-time systems are systems that have specific timing requirements. They are critical systems that play an important role in modern societies, be it for instance control systems in factories or automotives. In recent years, Ethernet has been increasingly adopted as layer 2 protocol in real-time systems. Indeed, the adoption of Ethernet provides many benefits, including COTS and cost-effective components, high data rates and flexible topology. The main drawback of Ethernet is that it does not offer "out-of-the-box" mechanisms to guarantee timing and reliability constraints. This is the reason why time-sensitive networking (TSN) mechanisms have been introduced to provide Quality-of-Service (QoS) on top of Ethernet and satisfy the requirements of real-time communication in critical systems. The promise of Ethernet TSN is the possibility to use a single network for different criticality levels, e.g, critical control traffic and infotainment traffic sharing the same network resources. This thesis is about the design of Ethernet TSN networks, and specifically about techniques that help quantify the extent to which a network can support current and future communication needs. The context of this work is the increasing use of design-space exploration (DSE) in the industry to master the complexity of designing (e.g. in terms of architectural and technological choices) and configuring a TSN network. One of the main steps in DSE is performing schedulability analysis to conclude about the feasibility of a network configuration, i.e., whether all traffic streams satisfy their timing constraints. This step can take weeks of computations for a large set of candidate solutions with the simplest TSN mechanisms, while more complicated TSN mechanisms will require even longer time. This thesis explores the use of Artificial Intelligence (AI) techniques to assist in the design of TSN networks by speeding up the DSE. Specifically, the thesis proposes the use of machine learning (ML) as an alternative approach to schedulability analysis. The application of ML involves two steps. In the first step, ML algorithms are trained with a large set of TSN configurations labeled as feasible or non-feasible. Due to its pattern recognition ability, ML algorithms can predict the feasibility of unseen configurations with a good accuracy. Importantly, the execution time of an ML model is only a fraction of conventional schedulability analysis and remains constant whatever the complexity of the network configurations. Several contributions make up the body of the thesis. In the first contribution, we observe that the topology and the traffic of a TSN network can be used to derive simple features that are relevant to the network feasibility. Therefore, standard and simple machine learning (ML) algorithms such as k-Nearest Neighbors are used to take these features as inputs and predict the feasibility of TSN networks. This study suggests that ML algorithms can provide a viable alternative to conventional schedulability analysis due to fast execution time and high prediction accuracy. A hybrid approach combining ML and schedulability analyses is also introduced to control the prediction uncertainty. In the next studies, we aim at further automating the feasibility prediction of TSN networks with the Graph Neural Network (GNN) model. GNN takes as inputs the raw data from the TSN configurations and encodes them as graphs. Synthetic features are generated by GNN, thus the manual feature selection step is eliminated. More importantly, the GNN model can generalize to a wide range of topologies and traffic patterns, in contrast to the standard ML algorithms tested before that can only work with a fixed topology. An ensemble of individual GNN models shows high prediction accuracies on many test cases containing realistic automotive topologies. We also explore possibilities to improve the performance of GNN with more advanced deep learning techniques. In particular, semi-supervised learning and self-supervised learning are experimented. Although these learning paradigms provide modest improvements, we consider them promising techniques due to the ability to leverage the massive amount of unlabeled training data. While this thesis focuses on the feasibility prediction of TSN configurations, AI techniques have huge potentials to automate other tasks in real-time systems. A natural follow-up work of this thesis is to apply GNN to multiple TSN mechanisms and predict which mechanism can provide the best scheduling solution for a given configuration. Although we need distinct ML models for each TSN mechanism, this research direction is promising as TSN mechanisms may share similar feasibility features and thus transfer learning techniques can be applied to facilitate the training process. Furthermore, GNN can be used as a core block in deep reinforcement learning to find the feasible priority assignment of TSN configurations. This thesis aims to make a contribution towards DSE of TSN networks with AI

SPATIAL-SPECTRAL FUZZY K-MEANS CLUSTERING FOR REMOTE SENSING IMAGE SEGMENTATION

Spectral clustering is a clustering method based on algebraic graph theory. The clustering effect by using spectral method depends heavily on the description of similarity between instances of the datasets. Althought, spectral clustering has been significant interest in recent times, but the raw spectral clustering is often based on Euclidean distance, but it is impossible to accurately reflect the complexity of the data. Despite having a well-defined mathematical framework, good performance and simplicity, it suffers from several drawbacks, such as it is unable to determine a reasonable cluster number, sensitive to initial condition and not robust to outliers. In this paper, we present a new approach named spatial-spectral fuzzy clustering which combines spectral clustering and fuzzy clustering with spatial information into a unified framework to solve these problems, the paper consists of three main steps: Step 1, calculate the spatial information value of the pixels, step 2 applies the spectral clustering algorithm to change the data space from the color space to the new space and step 3 clusters the data in new data space by fuzzy clustering algorithm. Experimental results on the remote sensing image were evaluated based on a number of indicators, such as IQI, MSE, DI and CSI, show that it can improve the clustering accuracy and avoid falling into local optimum.